Method for tracking movable object, tracking device, and method for controlling shooting parameters of camera

ABSTRACT

The embodiments of the disclosure provide a method for tracking a movable object, a tracking device, and a method for controlling shooting parameters of a camera. The method includes: determining a first on duration of a camera of the tracking device, wherein the first on duration comprises a starting time and an ending time; determining a second on duration of a plurality of light emitting elements disposed on the movable object by adding a first guard time before the starting time and adding a second guard time after the ending time; turning on the light emitting elements based on the second on duration; and controlling the camera to capture a specific image of the light emitting elements in the first on duration and accordingly tracking the movable object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of U.S. ProvisionalApplication No. 63/105,366, filed on Oct. 26, 2020, and U.S. ProvisionalApplication No. 63/145,497, filed on Feb. 4, 2021. The entirety of eachof the above-mentioned patent applications is hereby incorporated byreference herein and made a part of this specification.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The disclosure generally relates to tracking mechanism, in particular,to a method for tracking a movable object, a tracking device and amethod for dynamically controlling shooting parameters of a camera.

2. Description of Related Art

Nowadays, the mechanisms used for tracking objects in systems such asaugmented reality (AR) system or virtual reality (VR) system generallyinclude inside-out tracking and outside-in tracking. In the inside-outtracking mechanism, the to-be-tracked object may be disposed with lightemitting elements for the camera on the head-mounted display to capture,and the pose of the to-be-tracked object may be accordingly determined.

In this case, if the shooting parameters of the camera are not properlydesigned, some issues may occur. For example, if the on durations of thecamera and the light emitting elements are not properly designed, eitherunnecessary power consumption may be introduced, or the pose of theto-be-tracked object cannot be accurately determined. For anotherexample, if the shooting parameters use for performing environmentdetection and object tracking are the same, either the pose of theto-be-tracked object cannot be accurately determined, or the environmentcannot be accurately detected.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to a method for trackinga movable object, a tracking device and a method for dynamicallycontrolling shooting parameters of a camera, which may be used to solvethe above technical problem.

The embodiments of the disclosure provide a method for tracking amovable object, adapted to a tracking device. The method includes:determining a first on duration of a camera of the tracking device,wherein the first on duration comprises a starting time and an endingtime; determining a second on duration of a plurality of light emittingelements disposed on the movable object by adding a first guard timebefore the starting time and adding a second guard time after the endingtime; turning on the light emitting elements based on the second onduration; and controlling the camera to capture a specific image of thelight emitting elements in the first on duration and accordinglytracking the movable object.

The embodiments of the disclosure provide a tracking device including acamera and a processor. The processor is coupled to the camera andconfigured to perform: determining a first on duration of the camera ofthe tracking device, wherein the first on duration comprises a startingtime and an ending time; determining a second on duration of a pluralityof light emitting elements disposed on the movable object by adding afirst guard time before the starting time and adding a second guard timeafter the ending time; turning on the light emitting elements based onthe second on duration; and controlling the camera to capture a specificimage of the light emitting elements in the first on duration andaccordingly tracking the movable object.

The embodiments of the disclosure provide a method for dynamicallycontrolling shooting parameters of a camera. The method includes:determining a plurality of time frames, wherein the time frames comprisea plurality of first time frames and a plurality of second time frames;controlling the camera of the tracking device to shoot a first imagewith a first exposure parameter in each of the first time frames andaccordingly performing an environment detection;

controlling the camera of the tracking device to shoot a second imagewith a second exposure parameter in each of the second time frames andaccordingly performing a first specific detection, wherein the secondexposure parameter is lower than the first exposure parameter.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention.

FIG. 1 shows a functional diagram of a tracking device and a movableobject according to an embodiment of the disclosure.

FIG. 2A shows a first combination of the first on duration and thesecond on duration according to an embodiment of the disclosure.

FIG. 2B shows a second combination of the first on duration and thesecond on duration according to an embodiment of the disclosure.

FIG. 3 shows a flow chart of the method for tracking the movable objectaccording to an embodiment of the disclosure.

FIG. 4 shows a third combination of the first on duration and the secondon duration according to an embodiment of the disclosure.

FIG. 5 shows a flow chart of the method for dynamically controllingshooting parameters of a camera according to an embodiment of thedisclosure.

FIG. 6 shows the time frames according to a first embodiment of thedisclosure.

FIG. 7 shows the time frames according to the second embodiment of thedisclosure.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the present preferredembodiments of the invention, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numbers areused in the drawings and the description to refer to the same or likeparts.

See FIG. 1, which shows a functional diagram of a tracking device and amovable object according to an embodiment of the disclosure. In FIG. 1,the tracking device 100 may be any device capable of tracking themovable object 199 by using any known tracking mechanisms. In someembodiments, the tracking device 100 may be a head-mounted display (HMD)used in an AR system or a VR system, and the movable object 199 may be acomponent of the AR/VR system that needs to be tracked, such as ahandheld controller, but the disclosure is not limited thereto.

In the embodiments of the disclosure, the tracking device 100 includes acamera 102 and a processor 104. In various embodiments, the camera 102could be any cameras having charge coupled device (CCD) lens,complementary metal oxide semiconductor transistors (CMOS) lens, but thedisclosure is not limited thereto.

The processor 104 is coupled to the camera 102, and may be, for example,a general-purpose processor, a special purpose processor, a conventionalprocessor, a digital signal processor (DSP), a plurality ofmicroprocessors, one or more microprocessors in association with a DSPcore, a controller, a microcontroller, Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, anyother type of integrated circuit (IC), a state machine, and the like.

In the embodiments of the disclosure, the processor 104 may control thecamera 102 to capture images of the movable object 199 and determine thepose of the movable object 199 accordingly. In some embodiments, themovable object 199 may be disposed with a plurality of light emittingelements (such as regular light emitting diodes (LED) and/or infraredLEDs).

In this case, the processor 104 may perform inside-out tracking based onthe light distributions of the light emitting elements in the imagescaptured by the camera 102.

In the embodiments of the disclosure, the camera 102 may be configuredto capture images in the corresponding on durations (referred to asfirst on durations). That is, the camera 102 would not capture/shoot inthe durations other than the first on durations. Similarly, the lightemitting elements on the movable object 199 have their own on durations(referred to as second on durations). In this case, the light emittingelements would not emit light in the durations other than the second ondurations.

In the embodiments of the disclosure, the first on durations and thesecond on durations need to (partially) overlap with each other for thecamera 102 to capture the lights of the light emitting elements.However, if the first on durations and the second on durations are notproperly designed, some undesired effects may be introduced.

See FIG. 2A, which shows a first combination of the first on durationand the second on duration according to an embodiment of the disclosure.In FIG. 2A, the first on duration 211 is longer than the second onduration 212. In this case, the long exposure resulted from the first onduration 211 would brighten the background in the captured images, suchthat the objects needless to be tracked would also be captured in theimages. Accordingly, the performance of tracking the movable object 199would be degraded. In addition, since the exposure duration of thecamera 102 is usually very short, it is difficult to precisely controlthe light emitting elements to emit lights in the second on duration212.

See FIG. 2B, which shows a second combination of the first on durationand the second on duration according to an embodiment of the disclosure.In FIG. 2B, the light emitting elements on the movable object 199 aredesigned to be always on. In this case, the first on duration 221 may bedesigned to be short, which prevents the background in the capturedimages from being unnecessarily brightened. However, the powerconsumption of the light emitting elements would be high.

To solve the above problem, the embodiments of the disclosure provide amethod for tracking the movable object, and detailed discussion would beprovided in the following.

See FIG. 3, which shows a flow chart of the method for tracking themovable object according to an embodiment of the disclosure. The methodof this embodiment may be executed by the tracking device 100 in FIG. 1,and the details of each step in FIG. 3 will be described below with thecomponents shown in FIG. 1. For better understanding the concept of thedisclosure, FIG. 4 would be used as an example, wherein FIG. 4 shows athird combination of the first on duration and the second on durationaccording to an embodiment of the disclosure.

In step S310, the processor 104 determines a first on duration 411 ofthe camera 102 of the tracking device 100. In the embodiment, theprocessor 104 may determine the first on duration 411 to a length thatwould not be too long to overly brighten the background in the capturedimages, e.g., 16 μs, but the disclosure is not limited thereto. In FIG.4, the first on duration 411 has a starting time T1 and an ending timeT2.

In step S320, the processor 104 determines a second on duration 412 ofthe light emitting elements disposed on the movable object 199 by addinga first guard time G1 before the starting time T1 and adding a secondguard time G2 after the ending time T2. In various embodiments, thelengths of the first guard time G1 and the second guard time G2 may bearbitrarily chosen based on the requirements of the designer.

Next, in step S330, the processor 104 turns on the light emittingelements based on the second on duration 412. In one embodiment, if thelight emitting elements on the movable object 199 can be independentlycontrolled, the processor 104 can directly or indirectly control thelight emitting elements to emit lights in the second on duration 412. Inone embodiment, if the light emitting elements are powered andcontrolled by the movable object 199, the processor 104 may control themovable object 199 to turn on the light emitting elements in the secondon duration 412 for emitting lights during the second on duration 412.

In one embodiment, the processor 104 may synchronize with the movableobject 199 and/or the light emitting elements to notify that the lightemitting elements should be turned on in the second on duration 412.

In step S340, the processor 104 controls the camera 102 to capture aspecific image of the light emitting elements in the first on duration411 and accordingly tracking the movable object 199.

In one embodiment, the processor 104 may determine the pose of themovable object 199 based on the light distribution of the light emittingelements in the specific image by using the inside-out trackingmechanism, but the disclosure is not limited thereto.

Unlike the scenario in FIG. 2A, since the second on duration 412 islonger than the first on duration 411, it would be easier to control thelight emitting elements to emit light in the second on duration 412. Inaddition, compared with FIG. 2B, the power consumption of FIG. 4 wouldbe much lower, which improves the battery life of the movable object 199and/or the light emitting elements.

In addition, as mentioned in the above, if the shooting parameters usefor performing environment detection and object tracking are the same,either the pose of the to-be-tracked object cannot be accuratelydetermined, or the environment cannot be accurately detected. Therefore,the disclosure further proposes a method for dynamically controllingshooting parameters of a camera, which may be used to solve thisproblem.

See FIG. 5, which shows a flow chart of the method for dynamicallycontrolling shooting parameters of a camera according to an embodimentof the disclosure. The method of this embodiment may be executed by thetracking device 100 in FIG. 1, and the details of each step in FIG. 5will be described below with the components shown in FIG. 1.

In step S510, the processor 104 determines a plurality of time frames,wherein the time frames comprise a plurality of first time frames and aplurality of second time frames.

In step S520, the processor 104 controls the camera 102 of the trackingdevice 100 to shoot a first image with a first exposure parameter ineach of the first time frames and accordingly performing an environmentdetection.

In step S530, the processor 104 controls the camera 102 of the trackingdevice 100 to shoot a second image with a second exposure parameter ineach of the second time frames and accordingly performing a firstspecific detection, wherein the second exposure parameter is lower thanthe first exposure parameter.

For better understanding the concept of FIG. 5, FIG. 6 and FIG. 7 wouldbe used as example for discussions.

See FIG. 6, which shows the time frames according to a first embodimentof the disclosure. In FIG. 6, the processor 104 may divide the time axisinto a plurality of time frames, which are shown as rectangle boxes. Inthe present embodiment, the time frames may be divided into a pluralityof first time frames F11-F1N and a plurality of second time framesF21-F2N.

In one embodiment, the first time frames F11-F1N and the second timeframes F21-F2N are interleaved with each other. That is, a (2i+1)-thtime frame of the time frames belongs to the first time frames F11-F1N,and a (2i+2)-th time frame of the time frames belongs to the second timeframes F21-F2N, wherein i is an index.

For example, when i is 0, the (2i+1)-th time frame of the time framesmay be the 1-st time frame, which would be the first time frame F11. Inaddition, when i is 0, the (2i+2)-th time frame of the time frames maybe the 2-nd time frame, which would be the second time frame F21. Foranother example, when i is 1, the (2i+1)-th time frame of the timeframes may be the 3-rd time frame, which would be the first time frameF12. In addition, when i is 1, the (2i+2)-th time frame of the timeframes may be the 4-th time frame, which would be the second time frameF22.

In one embodiment, the processor 104 may control the camera 102 to shoota first image with the first exposure parameter in each of the firsttime frames F11-F1N and accordingly performing an environment detection.In one embodiment, the processor 104 may perform the environmentdetection based on a simultaneous localization and mapping (SLAM)mechanism, but the disclosure is not limited thereto.

In one embodiment, the first specific detection may be used for trackingthe movable object 199 disposed with the light emitting elements astaught in the above.

In one embodiment, since the processor 104 needs more feature points ofthe environment to better detect the environment, the processor 104 usesa higher first exposure parameter (e.g., higher exposure value and/orlonger exposure time) to guarantee that the environment could beproperly brightened in the first image. Accordingly, a betterperformance of the environment detection may be achieved in each of thefirst time frames F11-F1N.

On the other hand, when performing the first specific detection (e.g.,tracking the movable object 199), the processor 104 can use a lowersecond exposure parameter (e.g., lower exposure value and/or shorterexposure time) to dim the lights from the environment in the secondimage, such that only the lights from the light emitting elements wouldbe preserved in the second image. Accordingly, a better performance ofthe first specific detection may be achieved in each of the second timeframes F21-F2N.

In a second embodiment, the time frames may further include a pluralityof third time frames, and the processor 104 may control the camera 102to shoot a third image with a third exposure parameter in each of thethird time frames and accordingly performing a second specificdetection, wherein the third exposure parameter is higher than thesecond exposure parameter, and can be equal or not equal to the firstexposure parameter.

In one embodiment, the first specific detection is used for tracking themovable object 199, the second specific detection is used for thegesture detection or hand tracking.

See FIG. 7, which shows the time frames according to the secondembodiment of the disclosure. In FIG. 7, the processor 104 may dividethe timeline into a plurality of time frames, which are shown asrectangle boxes. In the present embodiment, the time frames may bedivided into a plurality of first time frames F11-F1N, a plurality ofsecond time frames F21-F2N, and a plurality of third time framesF31-F3N, wherein the first time frames F11-F1N, the second time framesF21-F2N, and the third time frames F31-F3N are interleaved with eachother.

That is, a (3i+1)-th time frame of the time frames belongs to the firsttime frames F11-F1N, a (3i+2)-th time frame of the time frames belongsto the second time frames F21-F2N, a (3i+3)-th time frame of the timeframes belongs to the third time frames F31-F3N, wherein i is an index.

For example, when i is 0, the (3i+1)-th time frame of the time framesmay be the 1-st time frame, which would be the first time frame F11.When i is 0, the (3i+2)-th time frame of the time frames may be the 2-ndtime frame, which would be the second time frame F21.

When i is 0, the (3i+3)-th time frame of the time frames may be the 3-rdtime frame, which would be the third time frame F31.

For another example, when i is 1, the (3i+1)-th time frame of the timeframes may be the 4-th time frame, which would be the first time frameF12. When i is 1, the (3i+2)-th time frame of the time frames may be the5-th time frame, which would be the second time frame F22. When i is 1,the (3i+3)-th time frame of the time frames may be the 6-th time frame,which would be the third time frame F32.

In one embodiment, the processor 104 may control the camera 102 to shoota first image with the first exposure parameter in each of the firsttime frames F11-F1N and accordingly performing an environment detection.In one embodiment, the processor 104 may perform the environmentdetection based on the SLAM mechanism, but the disclosure is not limitedthereto.

In one embodiment, the first specific detection may be used for trackingthe movable object 199 disposed with the light emitting elements astaught in the above.

In one embodiment, since the processor 104 needs more feature points ofthe environment to better detect the environment, the processor 104 usesa higher first exposure parameter to guarantee that the environmentcould be properly brightened in the first image. Accordingly, a betterperformance of the environment detection may be achieved in each of thefirst time frames F11-F1N.

On the other hand, when performing the first specific detection (e.g.,tracking the movable object 199), the processor 104 can use a lowersecond exposure parameter to dim the lights from the environment in thesecond image, such that only the lights from the light emitting elementswould be preserved in the second/third image. Accordingly, a betterperformance of the first specific detection may be achieved in each ofthe second time frames F21-F2N.

In summary, the embodiments of the disclosure can add guard timesbefore/after the first on duration of the camera of the tracking deviceto determine the second on duration of the light emitting elements, suchthat the light emitting elements can be easier controlled to emit lightin the second on duration while reducing the power consumption.

In addition, the embodiments of the disclosure can use a higher firstexposure parameter (e.g., higher exposure value and/or exposure time) toretrieve more feature points of the environment in the first image whenperforming environment detection in each of the first time frames.Accordingly, a better performance of the environment detection may beachieved in each of the first time frames.

On the other hand, the embodiments of the disclosure can use a lowersecond exposure parameter (e.g., lower exposure value and/or exposuretime) to make sure that only the lights from the light emitting elementswould be preserved in the second image. Accordingly, a betterperformance of the first specific detection may be achieved in each ofthe second time frames.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the structure of the presentinvention without departing from the scope or spirit of the invention.In view of the foregoing, it is intended that the present inventioncover modifications and variations of this invention provided they fallwithin the scope of the following claims and their equivalents.

What is claimed is:
 1. A method for tracking a movable object, adaptedto a tracking device, comprising: determining a first on duration of acamera of the tracking device, wherein the first on duration comprises astarting time and an ending time; determining a second on duration of aplurality of light emitting elements disposed on the movable object byadding a first guard time before the starting time and adding a secondguard time after the ending time; turning on the light emitting elementsbased on the second on duration; and controlling the camera to capture aspecific image of the light emitting elements in the first on durationand accordingly tracking the movable object.
 2. The method according toclaim 1, wherein the tracking device is a head-mounted display, and themovable object is a handheld controller connected to the head-mounteddisplay.
 3. The method according to claim 1, wherein the light emittingelements are infrared light emitting diodes.
 4. The method according toclaim 1, wherein the step of tracking the movable object comprises:determining a pose of the movable object based on a light distributionof the light emitting elements in the specific image.
 5. The methodaccording to claim 1, wherein before the step of controlling the movableobject to turn on the light emitting elements based on the second onduration, the method further comprises: synchronizing with the movableobject to notify that the light emitting elements should be turned on inthe second on duration.
 6. A tracking device, comprising: a camera; anda processor, coupled to the camera and configured to perform:determining a first on duration of the camera of the tracking device,wherein the first on duration comprises a starting time and an endingtime; determining a second on duration of a plurality of light emittingelements disposed on the movable object by adding a first guard timebefore the starting time and adding a second guard time after the endingtime; turning on the light emitting elements based on the second onduration; and controlling the camera to capture a specific image of thelight emitting elements in the first on duration and accordinglytracking the movable object.
 7. The tracking device according to claim6, wherein the tracking device is a head-mounted display, and themovable object is a handheld controller connected to the head-mounteddisplay.
 8. The tracking device according to claim 6, wherein the lightemitting elements are infrared light emitting diodes.
 9. The trackingdevice according to claim 6, wherein the processor performs: determininga pose of the movable object based on a light distribution of the lightemitting elements in the specific image.
 10. The tracking deviceaccording to claim 6, wherein before controlling the movable object toturn on the light emitting elements based on the second on duration, theprocessor further performs: synchronizing with the movable object tonotify that the light emitting elements should be turned on in thesecond on duration.
 11. A method for dynamically controlling shootingparameters of a camera, adapted to a tracking device, comprising:determining a plurality of time frames, wherein the time frames comprisea plurality of first time frames and a plurality of second time frames;controlling the camera of the tracking device to shoot a first imagewith a first exposure parameter in each of the first time frames andaccordingly performing an environment detection; controlling the cameraof the tracking device to shoot a second image with a second exposureparameter in each of the second time frames and accordingly performing afirst specific detection, wherein the second exposure parameter is lowerthan the first exposure parameter.
 12. The method according to claim 11,wherein the step of performing the environment detection comprises:performing the environment detection based on a simultaneouslocalization and mapping (SLAM) mechanism.
 13. The method according toclaim 11, wherein the first time frames and the second time frames areinterleaved with each other.
 14. The method according to claim 11,wherein a (2i+1)-th time frame of the time frames belongs to the firsttime frames, a (2i+2)-th time frame of the time frames belongs to thesecond time frames, wherein i is an index.
 15. The method according toclaim 11, wherein the first specific detection is used for tracking amovable object disposed with a plurality of light emitting elements, andthe method further comprises: determining a first on duration in each ofthe first time frames of the camera of the tracking device, wherein thefirst on duration comprises a starting time and an ending time;determining a second on duration of a plurality of light emittingelements disposed on the movable object by adding a first guard timebefore the starting time and adding a second guard time after the endingtime; controlling the movable object to turn on the light emittingelements based on the second on duration; and controlling the camera tocapture a specific image of the light emitting elements in the first onduration and accordingly tracking the movable object.
 16. The methodaccording to claim 11, wherein the time frames further comprise aplurality of third time frames, and the method further comprises:controlling the camera of the tracking device to shoot a third imagewith a third exposure parameter in each of the third time frames andaccordingly performing a second specific detection, wherein the thirdexposure parameter is higher than the second exposure parameter.
 17. Themethod according to claim 16, wherein first specific detection is usedfor tracking a movable object, the second specific detection is used fora gesture detection.
 18. The method according to claim 16, wherein thefirst time frames, the second time frames, and the third time frames areinterleaved with each other.
 19. The method according to claim 16,wherein a (3i+1)-th time frame of the time frames belongs to the firsttime frames, a (3i+2)-th time frame of the time frames belongs to thesecond time frames, a (3i+3)-th time frame of the time frames belongs tothe third time frames, wherein i is an index.